SIMSBIOHUB-374: BioHub Feature Submission Schema #209
Conversation
Remove DB_SCHEMA usage in most places. Remove DB Views Add Mock test data seed (disabled by default)
|
Openshift URLs for the PR Deployment: |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## dev #209 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.02%
==========================================
Files 213 214 +1
Lines 6093 6094 +1
Branches 938 938
==========================================
Hits 4205 4205
- Misses 1664 1665 +1
Partials 224 224 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Code looks good, but when I build I get the following error on the database container:
2023-11-15 23:52:30.133 PST [1] LOG: starting PostgreSQL 12.5 (Debian 12.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2023-11-15 23:52:30.133 PST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2023-11-15 23:52:30.133 PST [1] LOG: listening on IPv6 address "::", port 5432
2023-11-15 23:52:30.142 PST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-11-15 23:52:30.165 PST [85] LOG: database system was shut down at 2023-11-15 23:52:30 PST
2023-11-15 23:52:30.171 PST [1] LOG: database system is ready to accept connections
2023-11-15 23:52:55.132 PST [137] ERROR: permission denied for table user_identity_source
2023-11-15 23:52:55.132 PST [137] CONTEXT: SQL statement "select user_identity_source_id from user_identity_source where name = p_user_identity_source_name and record_end_date is null"
PL/pgSQL function api_set_context(character varying,character varying) line 21 at SQL statement
2023-11-15 23:52:55.132 PST [137] STATEMENT: select api_set_context($1, $2);
Good catch. I've pushed a change to fix this. The issue was that I assumed "GRANT ALL ON ..." would do the trick, but apparently that only works on currently existing items and doesn't work for items created afterwards. It now changes the default permissions for the biohub_api user, so it should have proper permissions for all future items as well. |
NickPhura
removed
the
Not Ready For Review
Add security tables/functions.
Update seed comment with latest sample query. ignore-skip
…ions, port user seed updates from sims
Improve old generated index names
|
@AlfredRosenthal Re Architectural Overview I should probably move this somewhere more formal, and maybe add some pictures, or something, but hopefully this helps a bit for now. The schema is basically divided into 4 chunks (not including users and messages, and other random tables): Data, Types/Properties, Search, Security. Data- submission (An entire submission record. Ie: a Survey). Types/PropertiesThese lookup tables will be used to provide context to the search_* fields, and also to provide our rules for validating incoming data. SearchThe search tables contain all of the key/value pairs parsed out of the submission_feature properties, based on the known/recognized properties for that feature type. The search tables are split into multiple tables so that the Security- security_rule (Contains a list of named rules as defined by the admins. Ex: 'Female Moose in Skeena') Similar to the search tables, the security condition tables are broken out into multiple tables so that each one can be strongly typed, and only worry about executing the rule against a specific type. Security Functions- evaluate_security_rule (Given the id of a submission_feature, runs all security rules against the feature, and returns those rules where ALL conditions returned true. This function runs all 4 of the below functions.) Each security condition function takes the name of a security condition rule, and the id of a submission_feature, and returns true if it applies. The functions themselves do a bit of dynamic sql generation allowing for a bit of query building for the rules (they read the comparator and value columns of the security_* row to generate sql). |
Add artifact security join table.
|
Kudos, SonarCloud Quality Gate passed!
|
* Submission Feature Migrations and Test Seeding Remove DB_SCHEMA usage in most places. Remove DB Views Add Mock test data seed (disabled by default)
* Submission Feature Migrations and Test Seeding Remove DB_SCHEMA usage in most places. Remove DB Views Add Mock test data seed (disabled by default)
* Submission Feature Migrations and Test Seeding Remove DB_SCHEMA usage in most places. Remove DB Views Add Mock test data seed (disabled by default)
Overview
Note: Includes breaking migration changes, so we will need to drop dev before merging this. Also, the api/app is likely going to be broken due to having removed lots of old database tables.
Links to Jira tickets
https://apps.nrs.gov.bc.ca/int/jira/browse/SIMSBIOHUB-374
Description of relevant changes
New migrations to add new feature/search/security/message tables. These tables are all net new. The one
feature_submissiontable does hang off the existingsubmissiontable.New see to populate the new lookup tables (with some realistic, but not final, values).
Removed all old biohub tables that we either know we won't need, or are likely to not need. I just updated the existing migrations, so we will need to do a clean wipe of dev before merging this PR.
Removed all code related to the database views, including the biohub_dapi_v1 user.
I also removed lots of references to
DB_SCHEMAas thebiohubschema may as well be hardcoded, given the earliest migrations all reference it directly anyways, so its not really configurable.Added a seed to insert mock test data (disabled by default).